﻿<HTML xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office">
<HEAD>
<META NAME="GENERATOR" Content="Microsoft Visual Studio 8.0">
<TITLE>Live Distributed Objects - Configuring Network Communication</TITLE>
<style type="text/css">
.style3 {
	color: #FF0000;
}
.style4 {
	margin-left: 40px;
}
</style>
</HEAD>
<BODY>
<h1>Live Distributed Objects</h1>
<h2>Configuring Network Communication</h2>
<p><span class="style3">Note: Contents of this tutorial may depend on code that has been written, 
debugged, or extended 5 minutes ago. If something doesn&#39;t work, make sure you&#39;re 
running the latest version of the platform. To update your installation without 
reinstalling everything, follow these
<a href="installation.html#incremental">
easy manual steps</a>, or configure an
<a href="installation.html#auto">automatic 
update</a>.</span></p>
<p><a href="#default">Default Communication</a><br>
&nbsp;&nbsp;&nbsp; <a href="#default_capabilities">Capabilities</a><br>
&nbsp;&nbsp;&nbsp; <a href="#default_configuration">Configuration</a><br>
<a href="#qsm">QuickSilver Scalable Multicast (QSM)</a><br>
&nbsp;&nbsp;&nbsp; <a href="#qsm_capabilities">Capabilities</a><br>
&nbsp;&nbsp;&nbsp; <a href="#qsm_configuration">Configuration</a><br>
&nbsp;&nbsp;&nbsp; <a href="#example_qsm">An Example Application Scenario Using 
QSM</a><br>
&nbsp;&nbsp;&nbsp; more... coming soon...<br>
<a href="#custom">Custom Transports</a><br>
&nbsp;&nbsp;&nbsp; coming soon... </p>
<h3><strong></a><a name="default"></a></strong>Default Communication</h3>
<p><strong><a name="default_capabilities"></a>Capabilities</strong></p>
<p>By default, the distribution is configured to use a simple transport based on 
a centralized server that can accept TCP connections from local or remote 
clients. The server maintains a global message queue and relays messages between 
clients. Communication is reliable and totally ordered, and state is transferred 
from existing to joining members in a manner synchronized with the stream of 
updates. While the server does relay messages, as configured it does not save 
data anywhere. No membership information is provided to the clients. Each client 
process keeps a single TCP connection to a known port on the server, for the 
purpose of exchanging data and checkpoints and passing control requests back and 
forth. Communication overhead does not depend on the number of communication 
channels in use, but the network overhead grows linearly with the number of 
clients per channel, and the overhead on the server grows linearly with the 
number of clients connected to the server. Due to the nature of the 
implementation (a multithreaded server with a single thread per client process), 
the system is not designed for large numbers of clients or very high data rates. 
The default communication mode is best suited for debugging, or for use with 
simple objects that have a low rate of updates. It is possible to configure a 
client machine to use multiple different servers, e.g. for different sets of 
channels.</p>
<p><strong><a name="default_configuration"></a>Configuration</strong></p>
<p>The server is implemented as a service, described and configured in file &quot;cc.liveobject&quot; 
in the &quot;.\services&quot; folder relative to the root installation path. The structure 
of the object is shown below.</p>
<p class="style4"><img src="screenshot_154.gif" width="1090" height="524"></p>
<p>The top level component is a general-purpose TCP server, which can accept 
client connections and associate them with communication channels generated by a 
subordinate channel factory object passed as an argument. The factory object is 
the actual server, which accepts TCP connections from clients. The server itself 
is configured in the registry, as discussed below. </p>
<p>The TCP controller uses two TCP ports. The first port, configured by the 
&quot;address&quot; parameter, is used to host a web service that clients invoke to 
negotiate connection parameters, such as encryption. The fault setting, as shown 
above, is to accept any clients on port 50000. The second port, configured by 
the &quot;mainport&quot; parameter, is used to accept the negotiated TCP connections from 
clients. By default, it is set to 60000. The second port is opened only on a 
single network adapter, selected by the &quot;network&quot; parameter that describes a 
network mask. By default, the mask is &quot;0.0.0.0/0&quot;, which means that the server 
will pick any network adapter it finds on the local machine. While this is 
appropriate for accepting clients from a local host, it should be changed to a 
meaningful value for network experiments. </p>
<p>For example, if the machine is connected to several networks, one of them 
being a wired LAN connection over which communication with clients should take 
place, the IP address of the LAN adapter is &quot;192.168.0.100&quot;, and the network 
mask is &quot;255.255.255.0&quot;, then the value of the subnet parameter could be 
&quot;192.168.0.0/24&quot; (meaning, the system should select the network adapter such 
that the&nbsp; most significant 24 bits of the address correspond to the prefix 
&quot;192.168.0&quot;), or it could be &quot;192.168.0.100/32&quot; (meaning, the system must select 
a network adapter with address exactly &quot;192.168.0.100&quot;). </p>
<p>Since both ports need to be accessible to clients, both need to be opened in 
the firewall.</p>
<p>As the above diagram suggests, the actual server is defined in the &quot;registry&quot; 
subfolder, under key &quot;5356357BBA5244009D3094BEBBB3A8B4&quot;. The structure of the 
object configured there by default is shown on the diagram below.</p>
<p class="style4"><img src="screenshot_155.gif" width="783" height="427"> </p>
<p>This default object acts as a server in a single-level hierarchy. All 
channels provided by this server as managed and configured locally. The object 
takes a single parameter &quot;root&quot;, which points to the path at which channels are 
stored, relative to the root of the installation path. By default, the server 
looks for channel configuration in the &quot;channels&quot; subfolder. The format of this 
folder has been discussed <a href="implementing.html">earlier</a>.</p>
<p>Normally, there is no need to modify the server object stored in the 
registry. However, one might replace the default server with an object that 
interfaces an external communication system, or connects to another, 
higher-level server, in a cascading manner, to address the scalability 
limitations of this centralized implementation. </p>
<p>On the client-side, channels are stored in a &quot;folder&quot; object configured in 
the registry. A typical channel will be simply a repository object, with the 
&quot;from&quot; clause pointing to a registry object shown below.</p>
<p class="style4"><img src="screenshot_156.gif" width="920" height="288"></p>
<p>For example, a shared text note in the examples folder has the following 
structure. Note how the above element appears in the &quot;from&quot; clause of the 
channel object.</p>
<p class="style4"><img src="screenshot_157.gif" width="1238" height="684"></p>
<p>As the diagram suggests, the client-side part of the communication system is 
configured in the registry, under key &quot;011E83FA6A9F426993BB8EF26FF7D7B8&quot;. The 
structure of the object stored there by default is shown below.</p>
<p class="style4"><img src="screenshot_158.gif" width="1212" height="520"></p>
<p>The client-side communication system is an object that is instantiated once 
in each client process, and then shared by all client objects in this process, 
and across all channels. The client-side object uses a single connection to 
communicate with the server (or in general, with its parent in a hierarchy). 
This connection is configured by the &quot;connection&quot; parameter. By default, the 
client is configured to open a TCP channel (the client-side equivalent of a TCP 
channel factory mentioned earlier) to a local server. </p>
<p>The client-side part of the TCP connection is configured using two 
parameters, &quot;address&quot; that points to the web service that controls connections 
on the server, and &quot;network&quot; that selects a specific network adapter. Both 
parameters should match their counterparts on the server, discussed earlier.</p>
<p>In the typical usage scenario, one would replace &quot;localhost&quot; in the value of 
the &quot;address&quot; parameter, both on the client side and on the server side with the 
actual DNS name or IP address of the server and replace the default &quot;0.0.0.0/0&quot; 
subnet selector with a value that corresponds to the physical medium over which 
communication should take place. One should then open both ports in use on the 
server and configure the &quot;channels&quot; folder on the server with channel metadata, 
as discussed earlier. </p>
<h3><a name="qsm"></a>QuickSilver Scalable Multicast (QSM)</h3>
<p><strong><a name="qsm_capabilities"></a>Capabilities</strong></p>
<p>QuickSilver Scalable Multicast (QSM) is an alternative to the default 
communication scheme where higher performance and scalability are desired, but 
total ordering and synchronized state transfer are not required, and where IP 
multicast is available. QSM provides reliable transfer, i.e. each transmitted 
message is delivered to all clients that are currently connected to the system 
despite possible losses in the network. Clients that join later will receive 
messages transmitted after they joined, but not messages transmitted earlier. 
The ordering of messages is FIFO, that is, messages from a given source are 
ordered with respect to one another, but messages from different sources are 
treated as if they belonged to separate sub-channels, and might be differently 
interleaved when they arrive at different recipients. Also, at this moment there 
is no state transfer support in QSM. In return for weaker ordering and the lack 
of state transfer, QSM provides greatly improved scalability (in our 
experiments, the engine experienced performance degradation at the level of 5% 
when scaled to 200 nodes), high data rates (QSM itself can transmit at network 
speeds), and very low overhead. Typical uses of QSM would be in settings where 
there is either one publisher, or a small number of publishers in combination 
with a locking scheme to determine who has the permission to transmit at the 
given time. QSM could be used in systems such as multimedia broadcasting, 
multicasting images or software updates etc., where one or a small number of 
servers for each channel distributed data to potentially a larger group of 
clients. Its reliability property is sufficient to ensure that the transmitted 
files or multimedia streams are delivered uncorrupted in their entirety. If 
clients need to receive streams starting from the beginning, one would typically 
use some form of broadcast controller protocol in combination with QSM, to 
schedule transmissions in a way such as to accommodate both existing clients 
(who need current data) and new clients (who need old data to catch up). If 
receiving old data is not needed and data is being continuously updated (as is 
the case e.g. when data represents recent values of a certain parameter, such as 
airplane coordinates published by a GMS device), then QSM could be used alone.</p>
<p>More information on the capabilities of QSM and its performance can be found 
in the following technical report.</p>
<p class="style4"><a href="http://hdl.handle.net/1813/9406">QuickSilver Scalable 
Multicast.</a><br>
Krzysztof Ostrowski, Ken Birman, Danny Dolev.<br>
Cornell University Technical Report, http://hdl.handle.net/1813/9406.</p>
<p>The current embedding of QSM into the live objects platform is initial and 
experimental, and it does impose a noticeable overhead, hence the performance 
achieved by QSM when used as a transport for live objects will be noticeably 
lower than when QSM is used alone. Currently, to get the best out of QSM, one is 
still better off using it simply as a library (see our website for the
<a href="http://www.cs.cornell.edu/projects/quicksilver/">original distribution</a>). 
We plan to gradually improve the quality of the embedding, but the ultimate 
future of QSM is to be subsumed and dissolved into its
<a href="http://www.cs.cornell.edu/~krzys/QuickSilver_JWSR.pdf">highly 
extensible</a> and
<a href="http://ecommons.library.cornell.edu/bitstream/1813/8221/1/TR2007-2088.pdf">
programmable</a> successor &quot;QS/2&quot;, which will eventually become the default 
multicast layer for the live objects platform. QS/2 is currently not available. 
We hope to release the initial prototype of QS/2 sometime later this year.</p>
<p><strong><a name="qsm_configuration"></a>Configuration</strong></p>
<p>The configuration of QSM involves three elements: (a) in-process client, (b) 
a system service, and (c) a remote controller, each of which runs a 
single-threaded QSM engine. The in-process client is a thin layer corresponding 
to the client part of the default communication substrate. A single instance of 
the in-process client lives in each process, and controls communication within 
the process. The client is connected using a single bidirectional link to a 
system service. The service can accept multiple clients, as it was the case with 
the default communication substrate. By default, the connection is done using a 
single TCP connection, and clients might be connected to a service that resides 
on a remote machine. However, the default model is to have a single system 
service on each machine to control local clients. While the current initial QSM 
wrapper uses TCP to connect clients to the system service, eventually we will 
provide a much more streamlined communication over shared memory, which will 
eliminate some memory copies and greatly reduce overhead thanks to non-blocking 
synchronization. In this target model, clients might not be able to access 
remote services (although we might leave the TCP option as an optional feature 
for selected clients). The shared memory connection will also be used to 
interface the platform to unmanaged applications.</p>
<p>Although some lowest-level components of QSM, such as scheduling and I/O, are 
used in each of the three elements above, the actual QSM protocol stack lives in 
a system service, i.e. by default there is a single instance of the QSM protocol 
stack running in a system service on each machine. While we might eventually 
support using QSM directly in a client process, currently this option is not 
available. Hosting QSM in a system service, as a local daemon controlling the 
entire node, is consistent with the design principles, QSM does not perform as 
well if more than a single instance of it is running on each&nbsp; node, and 
applications interfacing QSM directly need to written in a careful manner, so as 
not to block in callbacks from QSM, or promptly consume the received data. While 
hosting QSM in a system service adds a level of indirection, increases latency 
and the data path, for most casual uses this would still be the most appropriate 
solution (especially once we streamline the communication between client 
processes and the system service).</p>
<p>Instances of QSM running in system services on each node communicate over IP 
multicast. The services rely on a single remote controller component to 
administer the network. The controller maintains the list of communication 
channels and their configuration settings, and it hosts a global membership 
service that the QSM nodes use to agree on who&#39;s accessing the particular 
communication channels, and to configure the loss recovery protocol.</p>
<p>The in-process client is available as an element named &quot;Qsm&quot; from an object 
stored under registry key &quot;010BA5DB9C7F48B0A242A0AD2EF73C54&quot;, as shown below. As 
it was the case with the default communication substrate, the in-process client 
is modeled as a folder containing communication channels.&nbsp; </p>
<p class="style4"><img src="screenshot_159.gif" width="1137" height="506"></p>
<p>The usage is similar to what was shown earlier. For example, a text note 
hosted over a QSM channel would be constructed as shown below (the object is 
also available in the &quot;examples&quot; folder as a reference file &quot;shared_text_1.qsm&quot;). Note that this 
looks almost exactly as it did earlier, only that we replace the default 
&quot;Centralized Comm. Channel Client&quot; object with its equivalent &quot;QuickSilver 
Scalable Multicast&quot;. The object looks a bit more complicated because &quot;Qsm&quot; is 
not configured in the registry directly. Rather, it is an object provided by &quot;QuickSilver&quot;. 
The latter is present in multiple places in the registry, each occurrence is 
configured to play a different role. The instance stored at key 
&quot;010BA5DB9C7F48B0A242A0AD2EF73C54&quot; is configured to manage clients within an 
application process.</p>
<p class="style4"><img src="screenshot_160.gif" width="1340" height="836"></p>
<p>The in-process instance of QuickSilver that reisdes at key 
&quot;010BA5DB9C7F48B0A242A0AD2EF73C54&quot; and is referenced by the above looks as shown 
below. There is only one parameter here that is relevant, a &quot;configuration&quot;. The 
parameter itself is an extensible namespace containing values of different 
configuration settings. It can be edited in the object designer in the XML mode 
(you need to type valid XML as shown below for the designer to be able to parse 
the value).</p>
<p class="style4"><img src="screenshot_161.gif" width="741" height="500"></p>
<p>Each instance of QuickSilver, including the in-process instance that doesn&#39;t 
actually do any network communication, has a fairly number of configuration 
settings that can be tweaked and are discussed in another part of this tutorial. 
The settings that are most relevant are the following.</p>
<ul>
	<li><strong>qsm_application</strong> - a boolean value that should be 
	explicitly set to &quot;true&quot;, and which indicates that this instance of 
	QuickSilver can directly connect to clients</li>
	<li><strong>uplink_address</strong> - the address of the local service that 
	will manage this client, could be set to &quot;localhost&quot;</li>
	<li><strong>uplink_subnet</strong> - the network where the service is to be 
	found, could be &quot;0.0.0.0/0&quot;</li>
</ul>
<p>The settings of the &quot;uplink&quot; (connection to the local service) need to be 
coordinated with the matching settings in the &quot;configuration&quot; parameter of the 
service, as discussed below.</p>
<p>As mentioned earlier, the in-process client-side of QSM is a proxy that 
requires two server-side components to work. These are available as services in 
the &quot;services&quot; folder. The service to run locally on each client machine is 
stored in file &quot;quicksilver_localservice.liveobject&quot; and the service to run on 
one machine designated to manage the network is stored in file &quot;quicksilver_coordinator.liveobject&quot;. 
While the two might in theory run side-by-side, ideally only one of them would 
be hosted on a given machine. If the machine needs to have local clients and 
serve as a coordinator for QSM at the same time, one can merge the configuration 
settings to produce a single instance of QSM that plays both roles, as discussed 
below.</p>
<p>By default, the services mentioned above are &quot;commented out&quot; and each service 
file describes a null object reference. To &quot;activate&quot; the services, you need to 
remove a null reference and comment out the code. Note that while QSM services 
might run side by side with the default multicast service, they won&#39;t interact 
or relay messages to one another.</p>
<p>After un-commenting, the local system service described in &quot;quicksilver_localservice.liveobject&quot; 
has the following structure.</p>
<p class="style4"><img src="screenshot_162.gif" width="842" height="308"></p>
<p>The object definition stored in the registry looks as follows.</p>
<p class="style4"><img src="screenshot_163.gif" width="1151" height="542"></p>
<p>There are two sets of settings here that are relevant. The first is the 
&quot;configuration&quot; parameter of the QuickSilver instance, which we will discuss 
shortly. The second is the configuration of the channel that will be used to 
connect to the remote coordinator that QSM uses to manage the network. By 
default, the connection is done over a TCP channel. The configuration of the 
channel is similar as it was with communication between an in-process client 
component and the local service, which has already been discussed earlier.</p>
<p>An example set of configuration settings for the local QSM service is shown 
below. The relevant settings are discussed below.</p>
<p class="style4"><img src="screenshot_164.gif" width="697" height="987"></p>
<p>First, there are a few settings that describe what this instance of QSM will 
do.</p>
<ul>
	<li><strong>qsm</strong> - setting this value to &quot;true&quot; activates the QSM 
	protocol stack</li>
	<li><strong>asm_application</strong> - this value should be &quot;false&quot; because 
	in the default style of use, the local service does not directly accept 
	clients </li>
	<li><strong>qsm_controller</strong> - this is set to false because the local 
	instance of QSM on a client machine does not, by default, act as a 
	controller for the entire network of potentially many QSM clients</li>
</ul>
<p>These settings configure communication with in-process clients. In this 
version of the platform, we just hard-coded the communication between the 
in-process clients and the service to use the TCP connections mechanism 
discussed earlier. Eventually, we plan to replace it with shared memory 
communication. We may also support custom transports, but likely, we will mostly 
rely on a custom scheme because of the nature of the communication between these 
local processes (which would go over unmanaged memory and use some optimizations 
to reduce the amount of data being copied from one place to another on the 
critical path).</p>
<ul>
	<li><strong>uplink_controller</strong> - setting this to &quot;true&quot; allows this 
	instance of QuickSilver to accept client connections from the in-process 
	instances</li>
	<li><strong>uplink_address</strong> - this is similar as &quot;address&quot; in the 
	configuration of the TCP channel factory, and points to the address of a web 
	service that the system service will expose for in-process clients</li>
	<li><strong>uplink_port</strong> - likewise, this points to the second TCP 
	port used by the TCP channel factory to accept local in-process clients</li>
	<li><strong>uplink_subnet</strong> - used to select a network adapter, as 
	discussed earlier</li>
</ul>
<p>These settings are used to configure the QSM protocol stack.</p>
<ul>
	<li><strong>subnet</strong> - this selects the network adapter to use for 
	the actual communication, i.e. for the traffic where most messages will 
	flow; note that this must be an adapter on a network that supports IP 
	multicast; also, because by default in this experimental release we fix TTL 
	to 1, all the communicating nodes must be on the same LAN, and without any 
	routers in between </li>
	<li><strong>port</strong> - this is the main UDP port through which data 
	will flow in and out of QSM</li>
	<li><strong>gms_address</strong> - this is the address of the membership 
	service that will be hosted by the machine that acts as a coordinator for 
	the local network, and should match the settings used there</li>
</ul>
<p>Now, let&#39;s turn to the second service, which is to be used as a coordinator 
managing the network. After un-commenting, the service description looks as 
follows.</p>
<p class="style4"><img src="screenshot_165.gif" width="1088" height="558"></p>
<p>As mentioned earlier, the coordinator by default communicates with local 
services over TCP, and so at the top, we see the specification of the TCP 
channel factory. The configuration of such factory has already been discussed 
earlier. The actual object that produces communication endpoints to associate 
with the incoming TCP connections is again defined in registry. The registry 
definition looks as shown below.</p>
<p class="style4"><img src="screenshot_166.gif" width="708" height="461"></p>
<p>The only relevant parameter is the set of &quot;configuration&quot; settings. The 
values of &quot;qsm&quot; (set to true) and &quot;qsm_controller&quot; (also set to true) parameters 
indicate that this instance of QuickSilver runs the QSM protocol stack. However, 
this protocol stack by default won&#39;t (although it could) really be used in its 
entirety, only the part of it that is needed for the membership service will be 
active. Since the membership service communicates with its subordinates over 
UDP, using the same ports and the same network, we still include&nbsp; the 
&quot;subnet&quot; and &quot;port&quot; parameters. Other parameters are not needed here, as they 
relate to the actual multicast traffic that won&#39;t flow through this instance of 
QSM.</p>
<p>The discussion so far was rather abstract. In the next section, we shall 
describe the process of configuring QSM step by step for a simple application 
scenario, on a sample network with a few nodes.</p>
<p><strong><a name="example_qsm">An Example Application Scenario Using QSM</a></strong></p>
<p>coming soon.................................................................</p>
<h3>Custom Transports</h3>
<p>coming soon...</p>
</body>
</html>
